Skip to content

[BUG] Handle optional preamble lines in DepmtxReader and pad csc indptr#534

Open
sarapelka-blyk wants to merge 4 commits intoCORE-GATECH-GROUP:mainfrom
sarapelka-blyk:fix-depmatrix-reader
Open

[BUG] Handle optional preamble lines in DepmtxReader and pad csc indptr#534
sarapelka-blyk wants to merge 4 commits intoCORE-GATECH-GROUP:mainfrom
sarapelka-blyk:fix-depmatrix-reader

Conversation

@sarapelka-blyk
Copy link

@sarapelka-blyk sarapelka-blyk commented Sep 10, 2025

Closes #533

Make sure you have read over the developer guide to ease
the review process. These include:

Include a thorough and concise overview of the changes, and why this PR is overall beneficial to the project.

The previous version of the reader errors out when it comes across a preamble/initialization of the N0, N1, A, and Z matrices. Additionally it cannot handle a normalization factor at the top of the file.
Changes made:
Handle optional preamble lines in DepmtxReader (flx, N0/ZAI/N1)

  • Add support for optional flx = ... normalization factor
  • Allow skipping N0 = zeros(...), ZAI = zeros(...), N1 = zeros(...) initializers
  • Pad CSC indptr when depmtx has trailing all-zero columns
  • Store flux in DepmtxReader.flx

@sarapelka-blyk sarapelka-blyk changed the title [BUG] Handle optional preamble lines in DepmtxReader and pad csc indptr [BUG] Handle optional preamble lines in DepmtxReader and pad csc indptr Sep 10, 2025
@sarapelka-blyk sarapelka-blyk marked this pull request as ready for review September 10, 2025 08:25
@drewejohnson drewejohnson self-requested a review September 30, 2025 14:16
@drewejohnson
Copy link
Collaborator

Thanks @sarapelka-blyk! And sorry for the delay. I'll get on this this week

@drewejohnson drewejohnson linked an issue Oct 4, 2025 that may be closed by this pull request
Copy link
Collaborator

@drewejohnson drewejohnson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This behavior would really use some testing. I know the depletion files are lacking in functionality.

Could you create a copy of the existing depletion matrix file and add these preamble / zeros init behavior? Then we can show we get what we expect

Comment on lines +163 to +174
# Pad column pointer if file omits trailing all-zero columns
_, nCols = matrixSize
indptr = cscProcessor.indptr
expected = nCols + 1
if indptr.shape[0] < expected:
pad = expected - indptr.shape[0]
indptr = concatenate([indptr, full(pad, indptr[-1], dtype=indptr.dtype)])
elif indptr.shape[0] > expected:
raise ValueError(f"index pointer size ({indptr.shape[0]}) should be ({expected})")
self.depmtx = csc_matrix(
(cscProcessor.data[:, 0], cscProcessor.indices,
cscProcessor.indptr), dtype=longfloat, shape=matrixSize)
(cscProcessor.data[:, 0], cscProcessor.indices, indptr),
dtype=longfloat, shape=matrixSize)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This logic isn't immediately clear to me. If the shape of the matrix is known ahead of construction, does it matter that there are full zero columns? They shouldn't have any information in the sparse matrix structure.

Can you add some testing for this logic?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I ran it originally I got an error. As I understand it, scipy.sparse.csc_matrix expects indptr to be length n_cols + 1, even when some columns are all zero. In the depmtx files the matrix size includes columns that never appear in the A matrix, so those columns are implicit zeros. Padding indptr just makes the sparse structure consistent with the declared shape and avoids the pointer-size error.

BaseReader.__init__(self, filePath, 'depmtx')
SparseReaderMixin.__init__(self, sparse)
self.deltaT = None
self.flx = None
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar comments from #512

  1. Please add this to the Attributes section of the docstring, when it may exist or not, what purpose it serves, etc.
  2. Please consider using the full name self.flux

@sarapelka-blyk
Copy link
Author

My apologies, for taking a while! I have not forgotten about this, but I have been overwhelmed with work. I expect to be able to get back to this in December.

Copy link
Collaborator

@drewejohnson drewejohnson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the updates @sarapelka-blyk! I have some minor comments and then this will be ready to rock

Comment on lines +382 to +384
# optional flux captured
if hasattr(self.reader, "flx"):
self.assertIsInstance(self.reader.flx, float)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The attribute is now named flx so this will need an update. If we just want to make sure we have something, we can instead use

Suggested change
# optional flux captured
if hasattr(self.reader, "flx"):
self.assertIsInstance(self.reader.flx, float)
# optional flux captured
self.assertIsNotNone(self.reader.flux)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changed!

self.assertAlmostEqual(A[1, 1], 2.0) # A(2,2)
self.assertAlmostEqual(A[2, 1], -2.0) # A(3,2)

def test_sparse_indptr_invariants_if_sparse(self):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would greatly appreciate a docstring here as to what this test is accomplishing. At a certain point, it feels like we're testing the behavior of the scipy sparse arrays and I'm not sure how much value that adds.
But, the value is we want to make sure the indptr stops advancing for the last two columns. That's captured in the comments, but I would like it emphasized and stated in the docstring

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added blurb, let me know if you were thinking of something more detailed

Comment on lines +359 to +363
fd, cls._tmp = tempfile.mkstemp(suffix=".m", prefix="depmtx_optional_")
os.close(fd)
with open(cls._tmp, "w", encoding="utf-8") as f:
f.write(content)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm would using a NamedTemporaryFile be easier? Then we don't need to worry about the file descriptor object nor calls to os.close. Something like

@classmethod
def setUpClass(cls):
    content = """..."""
    cls._tmp = tempfile.NamedTemporaryFile(mode="w")
    cls._tmp.write(content)
    # Set file pointer back to start of file so we read from the top
    cls._tmp.seek(0)
...
@classmethod
def tearDownClass(cls):
    cls._tmp.close()

This may still require the try/except pattern during removal but I'm not sure.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changed. because i'm on windows i have set the delete flag to False (per https://docs.python.org/3/library/tempfile.html ), but that should be okay with linux and macs?

"""Base that writes a tiny depmtx file with optional preambles present."""

USE_SPARSE = True
__test__ = False
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm this is new to me!1

Since Pytest 2.6, users can prevent pytest from discovering classes that start with Test by setting a boolean __test__ attribute to False.

Looking at the two concrete classes below, could we have the "base" class be SparseOptionalPreambleTester and a subclass for dense? The only difference in the two is handling of the sparsity. Then we have two classes and don't need to modify the __test__ attribute

Footnotes

  1. https://docs.pytest.org/en/stable/example/pythoncollection.html

drewejohnson
drewejohnson previously approved these changes Jan 9, 2026
Copy link
Collaborator

@drewejohnson drewejohnson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @sarapelka-blyk!

@drewejohnson drewejohnson enabled auto-merge (squash) January 9, 2026 15:50
@drewejohnson
Copy link
Collaborator

Hmm looks like our CI is very outdated python runs: 3.6-3.9

I'll take a look hopefully soon and get that resolved elsewhere. Nothing for you to do, this can be considered complete 🚀

@drewejohnson
Copy link
Collaborator

Hmm looks like our CI is very outdated python runs: 3.6-3.9

I'll take a look hopefully soon and get that resolved elsewhere. Nothing for you to do, this can be considered complete 🚀

Aha! The target branch is develop which is wildly out of date.

If you can rebase your branch off main and change the target branch to be main, things should work much better

auto-merge was automatically disabled January 15, 2026 09:23

Base branch changed by a user without write access

@sarapelka-blyk sarapelka-blyk changed the base branch from develop to main January 15, 2026 09:23
@sarapelka-blyk sarapelka-blyk dismissed drewejohnson’s stale review January 15, 2026 09:23

The base branch was changed.

@sarapelka-blyk
Copy link
Author

If you can rebase your branch off main and change the target branch to be main, things should work much better

This was an exercise in patience and taking slow breaths.
@drewejohnson this needs to be re-approved

drewejohnson
drewejohnson previously approved these changes Jan 26, 2026
@sarapelka-blyk
Copy link
Author

all tests pass on my machine :((( i will look into the problem

@sarapelka-blyk
Copy link
Author

Not sure what had happened, but somewhere along the way I must have made a file binary instead of a text file. It should be fixed now.

@sarapelka-blyk
Copy link
Author

I cannot recreate the failing tests because they are passing locally :(

@sarapelka-blyk
Copy link
Author

I haven't forgotten about this issue again I promise! I will need to dedicate some time to it when I am available, hopefully within a month!

@drewejohnson
Copy link
Collaborator

I haven't forgotten about this issue again I promise! I will need to dedicate some time to it when I am available, hopefully within a month!

No worries, @sarapelka-blyk! I have not been very responsive, and I don't love that.

You know how to reach me if you want to brain storm and/or if this becomes urgent. Here to help :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BUG in depmtx reader

2 participants